Search for: All records

Large language models (LLMs) are notoriously memory-intensive during training, particularly with the popular AdamW optimizer. This memory burden necessitates using more or higher-end GPUs or reducing batch sizes, limiting training scalability and throughput. To address this, various memory-efficient optimizers have been proposed to reduce optimizer memory usage. However, they face critical challenges: (i) reliance on costly SVD operations; (ii) significant performance trade-offs compared to AdamW; and (iii) still substantial optimizer memory overhead to maintain competitive performance. In this work, we identify that AdamW's learning rate adaptation rule can be effectively coarsened as a structured learning rate update. Based on this insight, we propose Approximated Gradient Scaling for Memory-Efficient LLM Optimization (APOLLO), which approximates learning rate scaling using an auxiliary low-rank optimizer state based on pure random projection. This structured learning rate update rule makes APOLLO highly tolerant to further memory reductions while delivering comparable pre-training performance. Even its rank-1 variant, APOLLO-Mini, achieves superior pre-training performance compared to AdamW with SGD-level memory costs. Extensive experiments demonstrate that the APOLLO series performs on-par with or better than AdamW, while achieving greater memory savings by nearly eliminating the optimization states of AdamW. These savings provide significant system-level benefits: (1) Enhanced Throughput: 3x throughput on an 8xA100-80GB setup compared to AdamW by supporting 4x larger batch sizes. (2) Improved Model Scalability: Pre-training LLaMA-13B with naive DDP on A100-80GB GPUs without system-level optimizations. (3) Low-End GPU Friendly Pre-training: Pre-training LLaMA-7B on a single GPU using less than 12 GB of memory with weight quantization. 

The history of astronomy has shown that advances in sensing methods open up new windows to the Universe and often lead to unexpected discoveries. Quantum sensor networks in combination with traditional astronomical observations are emerging as a novel modality for multimessenger astronomy. Here we develop a generic analysis framework that uses a data-driven approach to model the sensitivity of a quantum sensor network to astrophysical signals as a consequence of beyond-the-standard model (BSM) physics. The analysis method evaluates correlations between sensors to search for BSM signals coincident with astrophysical triggers, such as black hole mergers, supernovae, or fast radio bursts. Complementary to astroparticle approaches that search for particlelike signals (e.g., weakly interacting massive particles), quantum sensors are sensitive to wavelike signals from exotic quantum fields. This analysis method can be applied to networks of different types of quantum sensors, such as atomic clocks, matter-wave interferometers, and nuclear clocks, which can probe many types of interactions between BSM fields and standard model particles. We use this analysis method to carry out the first direct search utilizing a terrestrial network of precision quantum sensors for BSM fields emitted during a black hole merger. Specifically, we use the global network of optical magnetometers for exotic physics (GNOME) to perform a search for exotic low-mass field (ELF) bursts generated in coincidence with a gravitational-wave signal from a binary black hole merger (GW200311_115853) detected by LIGO/Virgo on the March 11, 2020. The associated gravitational wave heralds the arrival of the ELF burst that interacts with the spins of fermions in the magnetometers. This enables GNOME to serve as a tool for multimessenger astronomy. Our search found no significant events and, consequently, we place the first lab-based limits on combinations of ELF production and coupling parameters.